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MULTI-MODAL HANDWRITING RECOGNITION CORRECTION 
FIELD OF THE INVENTION 

[01] The present invention generally relates to systems, methods, and computer-readable 
media with computer-executable instructions stored thereon for processing electronic 
ink data in pen-based computing systems and/or other computing systems. 

BACKGROUND 

[02] Typical computer systems, especially computer systems using graphical user 
interfaces (GUIs), are optimized for accepting user input from one or more discrete 
input devices, such as a keyboard for entering text, and a pointing device, such as a 
mouse with one or more buttons, for operating the user interface. An example of such 
a GUI is the user interface for the Windows® computer operating system (available 
from Microsoft Corporation of Redmond, Washington). The ubiquitous keyboard and 
mouse interface provides for fast creation and modification of documents, 
spreadsheets, database fields, drawings, photos, and the like. 

[03] Recently, however, pen-based computing systems, such as tablet PCs and the like, 
have been increasing in popularity. In pen-based computing systems, user input 
advantageously may be introduced using an electronic "pen" or stylus (e.g., akin to 
writing with a pen or pencil on a piece of paper). Indeed, in at least some pen-based 
computing systems, all user input is capable of being entered and manipulated using 
an electronic pen input device, and the user interface is fully controllable using only 
the electronic pen. 

[04] As pen-based computing systems are becoming more popular, users are increasingly 
entering more data in the form of electronic ink. In pen-based systems such as the 
tablet PC, users can write on the tablet PC just like they would write on a piece of 
paper. In many instances, however, users wish to convert the original electronic ink 
data to machine-generated text, text suitable for use and manipulation by conventional 
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word processing programs and other application programs. The tablet PC has 
handwriting recognition capabilities that convert handwriting to text. While the 
handwriting recognition in systems like the tablet PC is very good, it is not foolproof. 

[05] Accordingly, there is a need in the art to provide an efficient, reliable correction 
mechanism for allowing a user to correct handwriting recognition errors that result 
from the conversion of electronic ink to text. Such systems and methods would be 
particularly useful in pen-based computing systems and other devices that accept 
input in the form of electronic ink and/or from an electronic pen or stylus and/or 
systems or devices that display or otherwise process electronic ink data. 



SUMMARY 

[06] Aspects of the present invention relate to systems and methods for processing 
electronic ink. Such systems and methods may include: (a) receiving electronic ink 
input {e.g., through an input device, such as a digitizer); (b) generating a list of 
machine-generated objects based on the electronic ink input, the list including a first 
machine-generated object and alternative machine-generated objects; (c) converting 
the electronic ink input to the first machine-generated object; (d) displaying the first 
machine-generated object; (e) receiving speech input; (f) converting the speech input 
to a second machine-generated object (e.g., using a speech recognition engine), 
wherein the second machine-generated object is one of the alternative machine- 
generated objects and the list of machine-generated objects functions as a dictionary 
used for converting the speech input; and (g) replacing the first machine-generated 
object with the second machine-generated object. The machine-generated objects 
may correspond to words, lines, and/or other groupings of machine-generated text. 

[07] In some aspects of the invention, a user may confirm that the second machine- 
generated object is correct; and the system will replace the first machine-generated 
object with the second machine-generated object. Systems and methods according to 
some aspects of the invention may display the list of alternative machine-generated 
object candidates to the first machine-generated object based on the electronic ink 



-2- 



Patent Application Atty. Docket No.: 003797.00672 

input. In one aspect, the list of alternative machine-generated object candidates 
includes object candidates generated by a handwriting recognition engine based on 
the electronic ink input and object candidates based on a statistical language model 
(e.g., unigram, bigram, trigram). 

[08] Additional aspects of the invention relate to computer-readable media including 
computer-executable instructions stored thereon for performing various methods 
and/or operating various systems, including systems and methods like those described 
above. 

[09] These and other novel advantages, details, embodiments, features and aspects of the 
present invention will be apparent to those skilled in the art from following the 
detailed description of the invention, the attached claims and accompanying drawings, 
listed herein, which are useful in explaining the invention. 



BRIEF DESCRIPTION OF THE DRAWINGS 

[10] The above and other objects, features, and advantages of the present invention will be 
readily apparent and fully understood from the following detailed description, taken in 
connection with the appended drawings, in which: 

[11] Figure 1 illustrates a schematic diagram of a general-purpose digital computing 
environment in which certain aspects of the present invention may be implemented; 

[12] Figure 2 illustrates a pen-based personal computing (PC) environment in which 
certain aspects of the present invention may be implemented; 

[13] Figure 3 shows an illustrative user interface utilized in a tablet PC input panel for 
handwriting recognition correction. 

[14] Figure 4 shows an illustrative user interface utilized in Microsoft Windows Journal 
for handwriting recognition correction. 
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[15] Figures 5 and 6 show an example flow diagram illustrating procedures followed in at 
least some examples of this invention. 

[16] Figure 7 depicts a screen with electronic ink data and the associated machine- 
generated text that resulted from handwriting recognition in an illustrative 
implementation of the present invention. 

[17] Figure 8 depicts a screen including a correction user interface with a text candidate 
generated by speech recognition for correcting a previously incorrectly recognized ink 
input according to an illustrative embodiment of the present invention. 

[18] Figure 9 depicts a screen including a correction UI after a user has requested a soft 
keyboard in connection with an illustrative embodiment of the present invention. 

[19] Figure 10 depicts a screen including a correction user interface with a drop down 
menu of alternative text candidates according to an illustrative implementation of the 
present invention. 

[20] Figure 1 1 depicts a screen showing how a parsing error may be corrected according to 
an illustrative embodiment of the present invention. 



DETAILED DESCRIPTION 

[21] As described above, aspects of the present invention relate to systems, methods, and 
computer-readable media for processing electronic ink data. The following 
description is divided into sub-sections to assist the reader. The sub-sections include: 
Terms; General Description of Various Aspects of the Invention; Example Hardware; 
Example Systems, Methods, and Computer-Readable Media According to the 
Invention; and Conclusion. 

I. TERMS 

[22] The following terms are used in this specification and, unless otherwise specified or 
clear from the context, the terms have the meanings provided below: 
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[23] "Pen" - Any type of user input device useful in entering ink into and/or otherwise 
manipulating or controlling an electronic document. The terms "pen" and "stylus" are 
used interchangeably in this specification. 

[24] "Pen-Down Event" - An event that is initiated at the time a pen contacts a digitizer. 
Typically, a pen-down event will end at the time the pen leaves the digitizer surface 
(also called a "pen-up event" in this specification). 

[25] "Computer-Readable Medium" means any available media that can be accessed by a 
user on a computer system. By way of example, and not limitation, "computer- 
readable media" may include computer storage media and communication media. 
"Computer storage media" includes volatile and nonvolatile, removable and non- 
removable media implemented in any method or technology for storage of 
information, such as computer-readable instructions, data structures, program 
modules or other data. "Computer storage media" includes, but is not limited to, 
RAM, ROM, EEPROM, flash memory or other memory technology; CD-ROM, 
digital versatile disks (DVD) or other optical storage devices; magnetic cassettes, 
magnetic tape, magnetic disk storage or other magnetic storage devices; or any other 
medium that can be used to store the desired information and that can be accessed by 
a computer. "Communication media" typically embodies computer-readable 
instructions, data structures, program modules or other data in a modulated data 
signal, such as a carrier wave or other transport mechanism, and includes any 
information delivery media. The term "modulated data signal" means a signal that 
has one or more of its characteristics set or changed in such a manner as to encode 
information in the signal. By way of example, and not limitation, communication 
media includes wired media, such as a wired network or direct- wired connection, and 
wireless media, such as acoustic, RF, infrared and other wireless media. 
Combinations of any of the above should also be included within the scope of 
"computer-readable media." 
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II. GENERAL DESCRIPTION OF VARIOUS ASPECTS OF THE 
INVENTION 

[26] Aspects of the present invention relate to systems and methods for processing 
electronic ink in pen-based computing systems and/or other computing systems that 
process, display, and/or otherwise handle electronic ink data. One aspect of the 
invention relates to use speech to correct handwriting recognition errors. In another 
aspect, a spoken word can be recognized based on alternatives suggested by a 
handwriting recognition engine and the recognized spoken word can replace the 
handwritten recognized word on a display. In a further aspect, a user interface is 
provided which accepts speech input and allows for in-place correction of machine- 
generated text based on handwriting recognition. 

[27] Aspects of the present invention relate to systems and methods for processing 
electronic ink. Such systems and methods may include: (a) receiving electronic ink 
input (e.g., through an input device, such as a digitizer); (b) generating a list of 
machine-generated objects based on the electronic ink input, the list including a first 
machine-generated object and alternative machine-generated objects; (c) converting 
the electronic ink input to the first machine-generated object; (d) displaying the first 
machine-generated object; (e) receiving speech input; (f) converting the speech input 
to a second machine-generated object (e.g., using a speech recognition engine), 
wherein the second machine-generated object is one of the alternative machine- 
generated objects and the list of machine-generated objects functions as a dictionary 
used for converting the speech input; and (g) replacing the first machine-generated 
object with the second machine-generated object. The machine-generated objects 
may correspond to words, lines, and/or other groupings of machine-generated text. 

[28] In some aspects of the invention, a user may confirm that the second machine- 
generated object is correct; and the system will replace the first machine-generated 
object with the second machine-generated object. Systems and methods according to 
some aspects of the invention may display the list of alternative machine-generated 
object candidates to the first machine-generated object based on the electronic ink 
input. In one aspect, the list of alternative machine-generated object candidates 
includes object candidates generated by a handwriting recognition engine based on 
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the electronic ink input and object candidates based on a statistical language model 
(e.g., unigram, bigram, trigram). 



[29] Additional aspects of the invention relate to computer-readable media including 
computer-executable instructions stored thereon for performing various methods 
and/or operating various systems, including systems and methods like those described 
above. 

III. EXAMPLE HARDWARE 

[30] Fig. 1 illustrates a schematic diagram of a general-purpose digital computing 
environment that can be used to implement various aspects of the present invention. 
In Fig. 1, a computer 100 includes a processing unit 1 10, a system memory 120, and a 
system bus 130 that couples various system components including the system 
memory 120 to the processing unit 110. The system bus 130 may be any of several 
types of bus structures including a memory bus or memory controller, a peripheral 
bus, and a local bus using any of a variety of bus architectures. The system memory 
120 may include read only memory (ROM) 140 and random access memory (RAM) 
150. 

[31] A basic input/output system 160 (BIOS), which contains the basic routines that help 
to transfer information between elements within the computer 100, such as during 
start-up, is stored in the ROM 140. The computer 100 also may include a hard disk 
drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk 
drive 180 for reading from or writing to a removable magnetic disk 190, and an 
optical disk drive 191 for reading from or writing to a removable optical disk 192, 
such as a CD ROM or other optical media. The hard disk drive 170, magnetic disk 
drive 180, and optical disk drive 191 are connected to the system bus 130 by a hard 
disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive 
interface 194, respectively. These drives and their associated computer-readable 
media provide nonvolatile storage of computer-readable instructions, data structures, 
program modules, and other data for the personal computer 100. It will be 
appreciated by those skilled in the art that other types of computer-readable media 
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that can store data that is accessible by a computer, such as magnetic cassettes, flash 
memory cards, digital video disks, Bernoulli cartridges, random access memories 
(RAMs), read only memories (ROMs), and the like, may also be used in the example 
operating environment. 

[32] A number of program modules can be stored on the hard disk drive 170, magnetic 
disk 190, optical disk 192, ROM 140, or RAM 150, including an operating system 
195, one or more application programs 196, other program modules 197, and program 
data 198. A user can enter commands and information into the computer 100 through 
input devices, such as a keyboard 101 and pointing device 102 (such as a mouse). 
Other input devices (not shown) may include a microphone, joystick, game pad, 
satellite dish, scanner, or the like. These and other input devices often are connected 
to the processing unit 110 through a serial port interface 106 that is coupled to the 
system bus 130, but they also may be connected by other interfaces, such as a parallel 
port, game port, or a universal serial bus (USB), and the like. Further still, these 
devices may be coupled directly to the system bus 130 via an appropriate interface 
(not shown). 

[33] A monitor 107 or other type of display device also may be connected to the system 
bus 130 via an interface, such as a video adapter 108. In addition to the monitor 107, 
personal computers typically include other peripheral output devices (not shown), 
such as speakers and printers. In one example, a pen digitizer 165 and accompanying 
pen or stylus 166 are provided in order to digitally capture freehand input. Although a 
connection between the pen digitizer 165 and the serial port interface 106 is shown in 
Fig. 1, in practice, the pen digitizer 165 may be directly coupled to the processing unit 
1 10, or it may be coupled to the processing unit 1 10 in any suitable manner, such as 
via a parallel port or another interface and the system bus 130 as is known in the art. 
Furthermore, although the digitizer 165 is shown apart from the monitor 107 in Fig. 1, 
the usable input area of the digitizer 165 may be co-extensive with the display area of 
the monitor 107. Further still, the digitizer 165 may be integrated in the monitor 107, 
or it may exist as a separate device overlaying or otherwise appended to the monitor 
107. 
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[34] The computer 100 can operate in a networked environment using logical connections 
to one or more remote computers, such as a remote computer 109. The remote 
computer 109 can be a server, a router, a network PC, a peer device or other common 
network node, and it typically includes many or all of the elements described above 
relative to the computer 100, although for simplicity, only a memory storage device 
111 has been illustrated in Fig. 1 . The logical connections depicted in Fig. 1 include a 
local area network (LAN) 112 and a wide area network (WAN) 113. Such 
networking environments are commonplace in offices, enterprise-wide computer 
networks, intranets, and the Internet, using both wired and wireless connections. 

[35] When used in a LAN networking environment, the computer 100 is connected to the 
local area network 112 through a network interface or adapter 1 14. When used in a 
WAN networking environment, the personal computer 100 typically includes a 
modem 1 1 5 or other means for establishing a communications link over the wide area 
network 1 13, such as the Internet. The modem 1 15, which may be internal or external 
to the computer 100, may be connected to the system bus 130 via the serial port 
interface 106. In a networked environment, program modules depicted relative to the 
personal computer 100, or portions thereof, may be stored in the remote memory 
storage device. 

[36] It will be appreciated that the network connections shown are examples and other 
techniques for establishing a communications link between the computers can be 
used. The existence of any of various well-known protocols such as TCP/IP, 
Ethernet, FTP, HTTP, UDP, and the like is presumed, and the system can be operated 
in a user-server configuration to permit a user to retrieve web pages from a web-based 
server. Any of various conventional web browsers can be used to display and 
manipulate data on web pages. 

[37] Although the Fig. 1 environment shows one example environment, it will be 
understood that other computing environments also may be used. For example, one 
or more examples of the present invention may use an environment having fewer than 
all of the various aspects shown in Fig. 1 and described above, and these aspects may 
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appear in various combinations and subcombinations that will be apparent to one of 
ordinary skill. 



[38] Fig. 2 illustrates a pen-based personal computer (PC) 201 that can be used in 
accordance with various aspects of the present invention. Any or all of the features, 
subsystems, and functions in the system of Fig. 1 can be included in the computer of 
Fig. 2. The pen-based personal computer system 201 includes a large display surface 
202, e.g., a digitizing flat panel display, such as a liquid crystal display (LCD) screen, 
on which a plurality of windows 203 is displayed. Using stylus 204, a user can select, 
highlight, and write on the digitizing display area and thereby enter electronic ink data 
into the system. Examples of suitable digitizing display panels include 
electromagnetic pen digitizers, such as pen digitizers available from Mutoh Co. (now 
known as FinePoint Innovations Co.) or Wacom Technology Co. Other types of pen 
digitizers, e.g., optical digitizers, may also be used. The pen-based computing system 
201 interprets gestures made using stylus 204 in order to manipulate data, enter text as 
electronic ink, and execute conventional computer application tasks, such as creating, 
editing, and modifying spreadsheets, word processing programs, and the like. 

[39] The stylus 204 may be equipped with buttons or other features to augment its 
capabilities. In one example, a stylus 204 could be implemented as a "pencil" or 
"pen", in which one end constitutes a writing portion and the other end constitutes an 
"eraser" end, and which, when moved across the display, indicates portions of 
electronic ink on the display that are to be erased. Other types of input devices, such 
as a mouse, trackball, keyboard, or the like also could be used. Additionally, a user's 
own finger could be used as an input device, e.g., for selecting or indicating portions 
of the displayed image on a touch-sensitive or proximity-sensitive display. 
Consequently, the term "user input device," as used herein, is intended to have a 
broad definition and encompasses many variations on well-known input devices. 

[40] In various examples, the system provides an ink platform as a set of COM 
(component object model) services that an operating system and/or an application 
program can use to capture, manipulate, and store ink. The ink platform also may 
include a mark-up language including a language like the extensible markup language 
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(XML). Further, the system may use DCOM (distributed component object model) as 
another implementation. Yet further implementations may be used including the 
Win32 programming model and the .Net programming model from Microsoft 
Corporation. These platforms are commercially available and known in the art. 

[41] In addition to use with full performance pen-based computing systems or 'tablet PCs" 
(e.g., convertible laptops or "slate" type tablet PCs), aspects of this invention can be 
used in conjunction with other types of pen-based computing systems and/or other 
devices that accept data as electronic ink and/or that accept process, or display 
electronic pen or stylus input, such as: hand-held or palm-top computing systems; 
personal digital assistants; pocket personal computers; mobile and cellular telephones, 
pagers, and other communication devices; watches; appliances; and any other devices 
or systems that include a monitor or other display device and/or a digitizer that 
presents printed or graphical information to users and/or allows input using an 
electronic pen or stylus or which can process electronic ink collected by another 
device (e.g., a conventional desktop computer that can process electronic ink collected 
by a tablet PC). 

[42] The invention now will be described in conjunction with the remaining figures, which 
illustrate various examples of the invention and information to help explain the 
invention. The specific figures and information contained in this detailed description 
should not be construed as limiting the invention. 

IV. EXAMPLE SYSTEMS, METHODS, AND COMPUTER-READABLE 
MEDIA ACCORDING TO THE INVENTION 

[43] As described above, aspects of the present invention relate generally to systems and 
methods for processing electronic ink in pen-based computing systems and other 
computing systems. 

[44] In some instances, handwriting recognition technology will not correctly identify the 
original ink words producing machine-generated text that does not correctly 
correspond to the handwritten ink. Users may desire to correct these errors. 
Handwriting recognition of electronic ink input and correction of machine-generated 
text is provided in, for example, the tablet PC input panel (TIP) found in the Tablet 
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PC of Microsoft Corporation, the "Convert Handwriting to Text" function in the 
actions drop down menu in Microsoft Windows Journal (Journal) and "Convert Ink to 
Text" function in a drop down menu in Microsoft OneNote (OneNote). TIP, Journal 
and Onenote each have different user interfaces for both calling the handwriting 
recognition engine and for correcting handwriting recognition errors. 

[45] Fig. 3 shows an illustrative user interface 300 for TIP in a pen-based computer 
system. TIP provides a writing pad 310 for the user to input data in electronic ink 
format. The electronic ink data 320 input to the writing pad 310 is converted to 
machine-generated text by a handwriting recognition engine in the processor of the 
computer system. In the example of Fig. 3, a user has written the phrase, "Hello 
World" in electronic ink on the writing pad 3 10 of the TIP user interface 300 of a pen- 
based computing system. The system collects the ink data as the user inputs it, and 
then displays the sentence in the user's handwriting in electronic ink 320 on the 
writing pad 310. The ink data can be collected in any suitable manner and stored in 
any suitable format, such as ink serialized format ("ISF"). After the electronic ink 
data is collected, the system automatically sends the data to the handwriting 
recognition engine, which will attempt to recognize the handwritten ink data and 
convert the ink data to machine-generated text. Once the handwriting recognition 
engine has processed the electronic ink input data, it returns the recognized text to the 
application program (and/or otherwise makes it available to the application program). 
In this example, the machine-generated text 330 is displayed in the window 340 of the 
open application, e.g., Microsoft Word. As shown, the electronic ink input data 320 
has erroneously been recognized and converted to the machine-generated text 330 
"Hello Wind". To correct the handwriting recognition error, a user can rewrite the 
word on the writing pad 310, i.e., input the electronic ink input data, and have the 
system recognize the word again. This correction mechanism may not work because 
rewriting the same phrase can result in another erroneous recognition. Alternatively, a 
user can correct the erroneous recognition using a physical or soft keyboard to input 
the correct word. 

[46] Fig. 4 shows an illustrative user interface 400 for Journal in a pen-based computer 
system. In Fig. 4, the user has input electronic ink data on separate lines on Journal 
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page 410 "Hello World" and "I am testing handwriting recognition". When the user 
desires to convert the handwritten electronic ink input data to machine-generated text, 
the user selects the ink to be converted and, from an "Actions" drop down menu, 
chooses the function of "Convert Handwriting to Text" from the menu. In this 
example, the ink data "World" 415 is being considered for correction. Responsive to 
the user choosing the function of "Convert Handwriting to Text" from the drop down 
menu in this example, the text correction dialog 420 is displayed on the user interface 
400. As shown, the electronic ink input data is recognized and converted and 
displayed in the converted text region 430 of the text correction dialog 420. The "to 
be corrected" electronic ink input data is displayed in the ink region 435 of the text 
correction dialog 420. Also, a list of possible alternatives for the electronic ink input 
data displayed in the ink region 435 is displayed in the alternative candidate list 440 
of the text correction dialog 420. To correct errors in the electronic ink input data, the 
user can select one of the alternatives from the alternative list 440 or enter the word 
by using a keyboard in TIP. Should the user wish to correct the text associated with 
other electronic ink input data, she can select the word she wants to correct by tapping 
on the word with the stylus in the converted text region 430. Responsive to the 
selection of another word, the electronic ink corresponding to the selected word will 
appear in the ink region 435 and alternative candidates will be displayed in the 
alternative candidate list 440. 

[47] In OneNote, the user selects the desired handwritten ink data to be converted and 
chooses the "Convert Ink to Text" function from a Tools drop down menu. In 
response to selection of this function, Onenote replaces the original ink with machine- 
generated text. To correct a word, a user can delete the erroneous word, rewrite the 
word and perform another ink to text conversion or the user can correct the erroneous 
word by using a keyboard, such as a soft keyboard. 

[48] Selection of the electronic ink data in the examples above could take place in any 
suitable or desired manner or as the result of any suitable or desired action. For 
example, touching a stylus on the display screen (a "pen-down event") will result in 
selection of the specific word at the location of the contact (e.g., if the stylus contacts 
the digitizer screen within the "bounding box" occupied by a word, this action will 
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result in selection of that word). In other examples, a stylus may be detectable by a 
digitizer while hovering adjacent to the digitizer's surface. For example, some styli 
are "active," which in this specification means that they transmit, reflect, alter, or 
otherwise modify energy, data, or other information transmitted by and/or detectable 
by a digitizer when the stylus is located near the digitizer surface. As examples, a 
stylus may include circuitry or other structure that receives energy emitted by the 
digitizer and/or alter magnetic or electronic fields transmitted by and/or sensed by the 
digitizer, such that the presence of the stylus near the digitizer surface can be sensed 
by the digitizer due to changes in detected magnetic or electric fields. As another 
example, a stylus may transmit or reflect data, electrical or magnetic fields, or other 
energy or information that is detectable by the digitizer, at least when located near the 
digitizer surface. Any suitable or desired way of sensing or detecting the presence of 
a stylus when it is located near a digitizer surface may be used. Although a stylus and 
digitizer combination may be designed to allow detection of a stylus at virtually any 
reasonable position with respect to the digitizer surface, in at least some conventional 
pen-based computing systems with active pens, the stylus is initially detectable when 
it is about one-half to two inches from the digitizer surface, and in some examples, 
when it is approximately a half inch to an inch from the digitizer surface. In such 
systems, if desired, selection of a word may result when the stylus hovers above the 
surface of the display screen, and the word immediately beneath or adjacent the tip of 
the stylus during the hover action will be the word selected (e.g., if the hover action 
occurs over a bounding box of a word, that word will be selected). 

[49] As can be seen from the above examples, the user can either correct the word in two 
independent alternatives. First, by deleting the erroneous word and rewriting with the 
hope that the handwriting recognition engine will properly recognize and convert the 
ink data. Obviously, this correction mechanism may not work because rewriting the 
same phrase can result in another erroneous recognition. Alternatively, the user can 
use a keyboard (e.g., a soft keyboard or physical keyboard) to correct a word. This 
requires the user to employ the keyboard, which may not always be convenient, much 
less desired. Accordingly, it would be helpful to provide another methodology to 
correct handwriting recognition/conversion errors. 
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[50] Fig. 5 provides an illustrative flow diagram describing an example process that can be 
used to correct a handwriting recognition error according to certain implementations 
of the invention. When the process starts at step S500, the pen-based system receives 
an electronic ink input. As described, ink data can be gathered in any suitable manner 
and in any suitable format without departing from the invention, including in manners 
and formats that are conventional and known to those skilled in the art (such as ISF 
format). In step S502, an ink parser, preferably part of the operating system 195, 
analyzes and classifies the electronic ink input data as handwriting or drawing data, 
and if handwriting data, analyzes and groups the electronic ink input as text, e.g., 
word, line and paragraph. For illustrative purposes, the present invention will be 
described in terms of the electronic ink input data being handwriting, though the 
invention is not so limited. 

[51] Once the original electronic ink input data has been collected, analyzed and classified 
as handwriting (optionally, at various times while the ink data is being entered), the 
ink parser can send the electronic ink data to a handwriting recognition engine (e.g., 
in one implementation the handwriting recognition in Microsoft WISP (Windows Ink 
Services Platform) found in the Windows XP Tablet PC Edition operating system). 
The handwriting recognition engine converts the electronic ink input to machine- 
generated text in step S504. In step S506, the system displays the machine-generated 
text corresponding to the electronic ink input on a display such as a Tablet PC display 
screen. In one implementation of steps S504 and S506 involving a Tablet PC, the user 
can select the 'Tools" drop down menu, followed by selecting the option "Convert 
Ink to Text" from the "Tools" menu. Responsive to the user commands, the electronic 
ink input is converted to machine-generated text, which is displayed together (e.g., on 
top of or adjacent to) with the electronic ink. Alternatively, the machine-generated 
text may be displayed in place of the electronic ink. Fig. 7 shows an example of how 
the information may be displayed in steps S506. In this example, the original 
handwriting is softened and the converted text is shown on top of the original 
handwriting. 

[52] After converting the electronic ink input to machine-generated text, the user may 
select a portion of the machine-generated text for correction in step S508. For 
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example, to select the portion of text, the user may tap the area on the display with the 
stylus where the text is shown or hover over the portion of text with the stylus. The 
portion selected for correction may correspond to a word, letter, phrase, or sentence. 
The following discussion will describe an illustrative implementation of the invention 
involving correction of a word. It will be appreciated by one skilled in the art that the 
invention is not so limited, and the principles applied to a word can be applied to 
other portions of text for correction. 

[53] When converting the electronic ink to text in step S504, the handwriting recognition 
also generates a list of alternative machine-generated text candidates, which might be 
correct candidates for the electronic ink input. For example, the electronic ink input 
may be recognized as the machine-generated text "I", but other probable alternatives 
machine-generated text candidates that may be returned include "1", "L" "!" "i" "1" 
or "Z" due to their string similarity to "I". In step S509, the alternative-machine 
generated text candidates obtained from handwriting recognition for the selected word 
are defined as the dictionary for speech recognition. At this point speech input may be 
enabled. 

[54] A correction user interface can be provided to the user in step S510. The correction UI 
may automatically be provided or may be provided in response to a user input 
requesting the correction UI. The correction UI may at least in part take the form of a 
graphical user interface, which may include the ink representation of the selected 
portion for correction and also can, but does not have to, display the alternative 
machine-generated text candidates. The alternative machine-generated text candidates 
can be displayed, for example as a drop down menu, at the request of the user (e.g., by 
tapping on a portion of the UI). Initially, the list of alternative machine-generated text 
candidates may include only alternative text candidates identified during the 
handwriting recognition process as defined in step 509. In this instance, the alternative 
text candidates identified during handwriting recognition can be used as the dictionary 
or vocabulary for speech recognition. In an alternative implementation, the alternative 
text candidates can include text candidates identified by a statistical language model 
with or without the alternative text candidates identified by the handwriting 
recognition engine to be used as the dictionary for speech recognition. The alternative 
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text candidates may be displayed to the user in an order or with some other type of 
representation (e.g., symbol next to the candidate) indicating a confidence measure 
that a respective alternative text candidate corresponds to the electronic ink input. 

[55] Also, the user may request (e.g., by a pen down event on a portion of the correction 
user interface) that the initial dictionary for speech recognition be expanded to include 
additional machine-generated text candidates. The user may make such a request 
when the correct candidate is not found in the list of alternative machine-generated 
text candidates identified during handwriting recognition. In one implementation, the 
vocabulary for speech recognition can be expanded to include additional alternative 
machine-generated text candidates obtained from the statistical language model (LM). 
In one implementation, the text candidates obtained from the LM may be displayed 
together with list of text candidates identified during handwriting recognition with the 
confidence level of each candidate being represented to the user, for example by 
ranking most likely candidate to least likely candidate from the top of the list to the 
bottom of the list. 

[56] An illustrative LM that may be used to generate the alternative text candidates is the 
n-gram LM. In the LM, items of a unigram, bigram and trigram may be contained. It 
will be appreciated to one skilled in the art that a myriad of LMs may be used to 
identify additional machine-generated text candidates and that the use of unigram, 
bigram and trigram is merely illustrative. Based on the machine-generated text 
candidate and LM candidate list, a Viterbi search, known to those skilled in the art, 
may be performed to identify a path with the highest LM score among the candidate 
lists. Each node in the path can be assumed to be the most reasonable candidate (e.g., 
word in a grouping of words such as a sentence). 

[57] Once the correction user interface (UI) has been presented to the user in step S510, 
several methodologies for correction of the machine-generated text candidate may be 
available including, among others, correction by speaking a text candidate, correction 
by input through a soft keyboard, and correction by selecting a text candidate from a 
displayed list of alternative text candidates. Each of these possibilities will be 
discussed. 
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[58] For correction by speaking a text candidate, the step S510 of providing the correction 
user interface (UI) can include providing a mechanism for accepting speech input. 
Speech input may be enabled by, for example, a user tapping the stylus on a 
microphone icon, such as shown in the correction UI shown in Fig. 8. A user can 
speak the correct word using a speech input element such as a headset or microphone 
(e.g., built in to the computer), which is associated with the pen-based computer. The 
user speaks the word for correction into a speech input element and the speech input 
is received at step S512. 

[59] In step S514, the speech input is converted into an alternative machine-generated text 
candidate, the candidate being present in the alternative machine-generated text 
candidates list, which serves as the dictionary for speech recognition. In one 
implementation involving a pen-based system from Microsoft Corporation, the speech 
recognition engine processes the speech input using the Microsoft SAPI (speech API) 
5.1 with context-free grammar (CFG). The CFG grammar uses the 
vocabulary/dictionary for speech recognition created from the output of the 
handwriting recognition engine. When the user speaks the correct word, the speech 
recognition engine selects the alternative machine-generated text candidate 
corresponding to the spoken word from the list of text candidates and replaces the 
incorrect word with the machine-generated text candidate recognized by the speech 
recognition engine in step S516. The machine-generated text candidate recognized by 
speech recognition can first be displayed, for example in the correction UI of Fig. 8 
prior to correction. Upon user confirmation, for example by tapping outside the 
correction UI in Fig. 8 (or acknowledging through speech or other input mechanism), 
the text candidate identified by speech recognition can replace the incorrect word in 
the text sequence on the display. In the example of Fig. 8, the incorrect text 
recognized is the letter "Z", and the user has spoken the text candidate "i". If 
necessary, the user could open the drop down menu by hovering or by way of tapping 
the stylus on the down arrow in the correction UI. The drop down menu might 
provide another alternative speech recognition text candidate such as a capital "I" 
which the user could then select causing the letter "Z" to be replaced with *T\ 
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[60] At step S534 the correction process may be completed although it should be 
understood that the process may be repeated or another methodology for correction 
may be used. 

[61] For correction by input through a soft keyboard, the step S510 of providing the 
correction UI provides a mechanism allowing a user to select a soft keyboard for 
correction in step S518. Responsive to a user selection of a soft keyboard, a soft 
keyboard is on the display in step S520. An illustrative example of a correction UI 
after the soft keyboard has been displayed is shown in Fig. 9. In the example of Fig. 8, 
the incorrect text recognized is the letter "Z" In steps S524, to correct the text, the 
user can tap on a soft key(s) with the stylus to input the correct text and then tap on 
area outside the correction UI to cause the text to be corrected by replacing the text 
with the soft key input. At this point the correction process has been completed at step 
S534. 

[62] For correction by selecting a text candidate from a displayed list of alternative text 
candidates, the correction UI in step S510 can display the list of alternative text 
candidates or provide an interface allowing the user to request display of the 
alternative text candidates in step S526. Responsive to user selection causing a 
dropdown menu to open, the alternative text candidates provided by the handwriting 
recognition engine can be displayed in step S528, such as in confidence order as 
shown in Fig. 10. From the list of alternative text candidates, in step S530 the user can 
select the correct text candidate by, for example, tapping the stylus on the correct text 
candidate. Selection however, may take other forms such as by speech recognition, 
for example by receiving speech input and converting the recognized speech to the 
alternative text candidate as described above in connection with correction by speech 
recognition. 

[63] Also, the step of displaying the list of alternative text candidates in step S528 can 
include expanding or replacing the list of text candidates by, for example, tapping on 
the down arrow button at the bottom of the correction UI shown in Fig. 10. In this 
instance, the LM may be activated and an additional list of alternative text candidates 
can be displayed including the alternative text candidates suggested by the LM. In one 
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implementation, the number of alternative text candidates in the list may include 
twenty, though the number of candidates may be selected to accommodate display 
real estate or to provide all the candidates which exceed a certain threshold 
confidence level. It should be noted that obtaining LM text candidates can be 
processing intensive and the number of candidates may be limited based on the 
processing time needed to generate the LM candidate list. 

[64] In step S532, to correct the text, the user can tap on or hover over the text candidate in 
the candidate list with the stylus to replace the incorrect text. It will be appreciated 
that selecting the text candidate in step S532 could involve the user speaking one of 
the alternative text candidates on the alternative text candidate list with or without the 
user confirming that the speech recognized input is correct to replace the incorrect 
text. At step S534 the correction process may be completed although it should be 
understood that the process may be repeated or another methodology for correction 
may be used. 

[65] Of course, Figs. 5 and 6 merely constitute an example of steps that may be taken in 
accordance with at least some examples of the invention. Those skilled in the art will 
recognize that the content of the specific steps may be changed, the steps may be 
performed in a different order, additional steps may be added, and/or existing steps 
may be omitted or combined without departing from the invention. 

[66] Parsing errors can occur when the handwriting recognition engine recognizes 
electronic ink input data as two or more words where the ink input data was intended 
to represent one word or when electronic ink input data is recognized as one word 
where the input was intended to represent more than one word. Implementations of 
the invention allow for correction of parsing errors. User can select the correct 
grouping of electronic input text data by, for example circling (e.g., defining a region 
encompassing) the appropriate text with the stylus to either separate or group the 
appropriate text. Thus, when two words have been recognized when the user intended 
the ink strokes of the recognized words to be a single word, the user may circle the 
ink strokes making up both words with the stylus to indicate that they should be 
grouped together and considered as one word. Similarly, if two words have been 
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recognized as one word the user can circle the ink strokes corresponding to a portion 
of the recognized word to indicate that the circled ink strokes in the corresponding 
recognized word and the remaining ink strokes in the corresponding recognized word 
should be recognized as separate words. 

[67] Fig. 11 shows a user interface where the written phrase "am testing" has been 
recognized as "antedating". In the Fig. 1 1 example, the user has circled the ink strokes 
with the stylus corresponding to the word "am". The selected ink strokes can be 
highlighted in some manner, such as shown in Fig. 11, once the user lifts the stylus 
from the digitizer. At this juncture, the user can provide an input (e.g., tapping the 
stylus on or hover over the highlighted portion, providing a voice command) causing 
the handwriting recognition engine to convert the selected ink strokes to text and the 
other non-highlighted input strokes (corresponding to "testing") to text. The machine- 
generated text candidate is displayed. The operation for grouping strokes is analogous 
in that the grouped strokes are highlighted, the user provides an input requesting the 
handwriting recognition engine to convert the strokes to text, and the resulting 
machine-generated text candidate is displayed. 

[68] Below is a more detailed explanation of applying the LM to generate the previously 
described candidate lists. Given a word sequence, 



EZ: l| w t _ 

















W i + 2 



the handwriting recognition engine gives each word w { some candidates. Assume the 

user wants to correct the word w i , a word list which the speech recognition engine can 

use must be generated. The idea is that the words in the dictionary can be filtered 
using the string similarity, and then the LM can be used to calculate the probability 
that the word appears in the context. Finally, the weighted sum of the score in the LM 
and the score in string similarity are used for ranking the words in the dictionary for 
the speech recognition engine. In one implementation, twenty candidates may be 
selected, though this number can vary. One factor that may influence the number of 
candidates is display real estate. 
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[69] The candidates for the word w i given by the handwriting recognition engine may not 

contain the correct word, but the sub segments of the correct word are usually the sub 
segments of these candidate words. Thus string similarity information may be used to 
determine candidates. A string matching process may be conducted using the 
Dynamic Programming String Alignment algorithm. For example, assume the word 
"these" and the word "horses", the result of string matching is as below. 



these 



1 




f i 


r 


horse s 



The count of the matched characters is 3. The candidates of w i given by the 
handwriting recognition engine are represented as w. j . To a word d k in the 
dictionary, the similarity between d k and w. . is defined as 

s s im (4 t , w. ) = log(— ' ) 

len(w u ) + len(d k ) 

c match( w ij>dk) i s the count of the matched characters between d k and w f J 
(considering order), the leniw^ J) is the character count of wordw i y , and the len(d k )is 
the character count of word d k . To the word "these" and the word "horses", 

s sim (" these" , " horses" ) = log(-^— ) « -0.56427 1 

5 + 6 

The score of d k in string similarity is 

^ sim 

[70] Theoretically speaking, every word d k in the dictionary is a candidate of w j for the 
speech recognition engine. But if the string similarity between d k and any is 
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very small, it is not necessary to use d k as the candidate of w i for the speech 

recognition engine, because it can be assumed that although the handwriting 
recognition engine cannot give the correct word, it can give some (enough) characters 
of the correct word. For example, if the candidate of w i that the handwriting 
recognition engine provides is "these", the word "is" is almost impossible to be the 
correct word of w i because the similarity between the two words is very small. 
Applying this method, the computing complexity can be reduced. Furthermore, 
because the count of the common characters between d k and any w u (not 

considering order) is equal to or greater than the count of the matched characters 
between d k and w. . , if there is no common character between d k and w f J or the 

count of the common characters is very small, the string similarity between d k and 
any w t J does not need to be calculated. 

[71] For each word d k in the dictionary following the string matching process, the LM 
model 
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can be applied to calculate the probability of occurrence of the word d h in the word 
sequence using the Language Model. 
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The score of the word d k in the LM is 

Sim (dk ) = log(P(w f ._ 2 w i _ x d h w M w M )) 

[72] To generate the word list for the speech recognition engine, the score of the LM and 
the score of the string similarity can be combined and weighted to obtain the weighted 
score, 

^(d k ) = s iM (d k ) + A* s sim (d k ) 

X is the weight of the score of string similarity. Using the weighted score, the 
confidence level associated with each word can be determined relative to other words 
and the words can be ranked based on the probability that they would correspond to 
the electronic ink input. In one implementation, the top twenty words may be selected. 

[73] In sum the steps for generating candidates using the LM is 1) the words in the 
dictionary are filtered using the string similarity score; 2) the LM score is calculated; 
and 3) the words are ranked using the weighted sum of the LM score and the string 
similarity score, and the top (e.g., twenty) ranked words are selected as machine- 
generated text candidates. 

[74] Those skilled in the art will recognize that aspects of this invention can be applied to 
any type of electronic ink data and machine-generated objects, including handwritten 
text in any language and written or read in any direction without departing from the 
invention. Additionally, aspects of this invention can be applied to recognition and 
processing of other types of data and machine-generated objects without departing 
from the invention, such as data and elements in charts, diagrams, graphs, flowcharts, 
etc.; musical data or symbols; mathematical or scientific data or symbols; drawings; 
etc. 

[75] Finally, the present invention also relates to computer-readable media including 
computer-executable instructions stored thereon for performing various methods 
and/or for use in various systems, including those systems and methods described 
above. The computer-readable media may constitute computer-executable instructions 
stored on the various specific examples of computer-readable media described above. 
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[76] Various examples of the present invention have been described above, and it will be 
understood by those of ordinary skill that the present invention includes within its 
scope all combinations and subcombinations of these examples. Additionally, those 
skilled in the art will recognize that the above examples simply exemplify various 
aspects of the invention. Various changes and modifications may be made without 
departing from the spirit and scope of the invention, as defined in the appended 
claims. 
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